Telegram Group & Telegram Channel
Telegram Group » Japan » Parallel Experiments » Telegram Webview » Post 938
Parallel Experiments
https://arxiv.org/abs/2305.18290 #llm #ai

今天深入学习了 DPO，再次感叹扎实的数学功底对 AI/ML Research 的重要性……

原始的 RLHF 是用 pairwise human preference data（A 和 B 哪个更好）去训练一个 reward model，然后用 RL 来训练主 policy model，objective 是 minimize negative log likelihood + regularization（比如 PPO 就是通过新旧 policy 之间的 KL Divergence 来做 regularization）。这样的缺点在于 RL 是出了名的难搞，而且还需要一个 critic model 来预测 reward，使得整个系统的复杂性很高。

DPO 的思路是，观察到 RLHF 的 objective 本质上是 minimize loss over (latent) reward function，通过一番 reparameterization 等数学推导，重新设计了一个 minimize loss over policy 的 objective，绕过了中间这个 reward model，让 gradient update 直接增加 policy model 生成 winner response 的概率并降低 loser response 的概率，大幅简化了流程。

拓展阅读：
- KTO: 更进一步，不需要 pairwise comparison，只用对 individual example 的 upvote/downvote 也可以学习到 preference。
- IPO: 解决 DPO 容易 overfit 的问题。
arXiv.org
Direct Preference Optimization: Your Language Model is Secretly a...
While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely...
www.tg-me.com/jp/Parallel Experiments/com.LinghaoCh/938
1.2K viewsLinghao Zhang, edited  



tg-me.com/LinghaoCh/938
Create:
Last Update:

https://arxiv.org/abs/2305.18290 #llm #ai

今天深入学习了 DPO，再次感叹扎实的数学功底对 AI/ML Research 的重要性……

原始的 RLHF 是用 pairwise human preference data（A 和 B 哪个更好）去训练一个 reward model，然后用 RL 来训练主 policy model，objective 是 minimize negative log likelihood + regularization（比如 PPO 就是通过新旧 policy 之间的 KL Divergence 来做 regularization）。这样的缺点在于 RL 是出了名的难搞，而且还需要一个 critic model 来预测 reward，使得整个系统的复杂性很高。

DPO 的思路是，观察到 RLHF 的 objective 本质上是 minimize loss over (latent) reward function，通过一番 reparameterization 等数学推导，重新设计了一个 minimize loss over policy 的 objective，绕过了中间这个 reward model，让 gradient update 直接增加 policy model 生成 winner response 的概率并降低 loser response 的概率，大幅简化了流程。

拓展阅读：
- KTO: 更进一步，不需要 pairwise comparison，只用对 individual example 的 upvote/downvote 也可以学习到 preference。
- IPO: 解决 DPO 容易 overfit 的问题。

BY Parallel Experiments




Share with your friend now:
tg-me.com/LinghaoCh/938

View MORE
Open in Telegram


Parallel Experiments Telegram | DID YOU KNOW?

Date: |

How Does Bitcoin Work?

Bitcoin is built on a distributed digital record called a blockchain. As the name implies, blockchain is a linked body of data, made up of units called blocks that contain information about each and every transaction, including date and time, total value, buyer and seller, and a unique identifying code for each exchange. Entries are strung together in chronological order, creating a digital chain of blocks. “Once a block is added to the blockchain, it becomes accessible to anyone who wishes to view it, acting as a public ledger of cryptocurrency transactions,” says Stacey Harris, consultant for Pelicoin, a network of cryptocurrency ATMs. Blockchain is decentralized, which means it’s not controlled by any one organization. “It’s like a Google Doc that anyone can work on,” says Buchi Okoro, CEO and co-founder of African cryptocurrency exchange Quidax. “Nobody owns it, but anyone who has a link can contribute to it. And as different people update it, your copy also gets updated.”

Can I mute a Telegram group?

In recent times, Telegram has gained a lot of popularity because of the controversy over WhatsApp’s new privacy policy. In January 2021, Telegram was the most downloaded app worldwide and crossed 500 million monthly active users. And with so many active users on the app, people might get messages in bulk from a group or a channel that can be a little irritating. So to get rid of the same, you can mute groups, chats, and channels on Telegram just like WhatsApp. You can mute notifications for one hour, eight hours, or two days, or you can disable notifications forever.

Parallel Experiments from jp


https://arxiv.org/abs/2305.18290 #llm #ai今天深入学习了 DPO，再次感叹扎实的数学功底对 AI/ML Research 的重要性……原始的 RLHF 是用 pairwise human preference data（A 和 B 哪个更好）去训练一个 reward model，然后用 RL 来训练主 policy model，objective 是 minimize negative log likelihood + regularization（比如 PPO 就是通过新旧 policy 之间的 KL Divergence 来做 regularization）。这样的缺点在于 RL 是出了名的难搞，而且还需要一个 critic model 来预测 reward，使得整个系统的复杂性很高。DPO 的思路是，观察到 RLHF 的 objective 本质上是 minimize loss over (latent) reward function，通过一番 reparameterization 等数学推导，重新设计了一个 minimize loss over policy 的 objective，绕过了中间这个 reward model，让 gradient update 直接增加 policy model 生成 winner response 的概率并降低 loser response 的概率，大幅简化了流程。拓展阅读：- KTO: 更进一步，不需要 pairwise comparison，只用对 individual example 的 upvote/downvote 也可以学习到 preference。- IPO: 解决 DPO 容易 overfit 的问题。

 Parallel Experiments TG
Webview: 938
Parallel Experiments.Telegram Webview
Parallel Experiments Telegram TG Channel
Telegram Updated:
Telegram Parallel Experiments
FROM USA